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Abstract 

Engagement in Massive Open Online Courses (MOOCs) is based on students who self-organize 
their participation according to their own goals and interests. Visual materials such as videos and 
discussion forums are basic ways of engaging students in MOOCs. Student achievement in 
MOOCs is typically measured using assessments distributed throughout the course. Although there 
is research on the basic forms of student’s engagement and assessment in MOOCs, little is known 
about their effect on students’ achievement in the form of students completing a MOOC. Using 
binomial logistic regression models, this paper addresses this gap in the literature by presenting 
the degree to which student engagement with videos and forum posts can predict students’ 
probability of achievement in a MOOC. It also explores the extent to which participation behaviors 
and their intention to receive the course certification can be used to predict achievement in 
MOOCs. Using qualitative content analysis, this paper discusses the quality of the forum posts 
exchanged by participants in this MOOC. The findings from quantitative analysis support 
MOOC’s pedagogical assumptions, showing that students’ engagement in forums and with videos 
increases the probability of course achievement. It also shows that intention to certify plays a 
moderator effect on the number of videos watched, enhancing achievement in MOOCs. The 
findings from qualitative analysis reveal that most students’ posts in forums display more 
information acquisition than critical thinking. Implications for practice suggest MOOC designers 
and MOOC instructors foster engagement in forums by implementing discussion prompts that 
foster interactions about deep meaning of concepts or application of concepts covered in the 
MOOC. In regard to videos, implications for practice suggest the creation of interactive videos 
that promote students’ engagement and control such as inserting guiding questions and segmenting 
the video content. Future research comprising multiple MOOC cohorts is suggested to validate the 
empirical model presented in this study. 
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How Much Does Student Engagement with Videos and Forums 
in a MOOC Affect Their Achievement? 

Massive Open Online Courses (MOOCs) are open learning environments that have the 
capacity to enroll a large number of participants. In MOOCs, participants freely engage and 
disengage with available resources and share their learning experiences with other participants in 
discussion forums. MOOC resources are commonly comprised of online reading materials, videos, 
quizzes, discussion forums, and assessments. Due to their open nature and students’ autonomy, 
MOOCs are also known for having a high number of students drop out (Ho et ah, 2014; Coetzee, 
Fox, Hearst, & Hartmann, 2014). The combination of students’ autonomy and students’ dropout 
rates has drawn the attention of researchers for better understanding the forms of students’ 
engagement and its contribution to students’ achievement in MOOCs (e.g., Onah, Sinclair, & 
Boyatt, 2014). 

In this paper, we explore the effect of students’ intention to complete the MOOC and their 
engagement with videos and forums on their achievement in the Creativity, Innovation and Change 
MOOC (CIC MOOC) delivered by The Pennsylvania State University. Binomial logistic 
regression models were used to present the degree to which learner’s engagement with videos and 
forum posts can be the basis for predicting course completion and receiving a certificate. We use 
qualitative content analysis to gain knowledge about the quality of the forum posts exchanged by 
participants in this study. For that, we make use of the Interaction Analysis Model (IAM) provided 
by Gunawardena, Lowe, and Anderson (1997) to evaluate participants’ online messages in terms 
of co-construction of knowledge. Afterwards, we discuss the results through the lens of the 
literature and indicate implications for practice and research. 


Review of Related Literature 

The most common ways to engage students in MOOCs are insertion of visual materials 
such as videos and the use of discussion forums. Formative assessments (low or no point value 
quizzes) and summative assessments (graded evaluations) are often used to evaluate participants 
performance in MOOCs. This section presents a brief review of the literature about students’ 
engagement with videos and forums, students’ completion and their assessment in MOOCs. 

Student Engagement with Videos in MOOCs 

The majority of MOOCs delivered through platforms such as Coursera, edX, and Udacity 
heavily rely on videos to deliver course content. Videos give flexibility to participants and allow 
scalability for MOOC providers in delivering course content (Lee et ah, 2015). These videos vary 
in length, in position within the course, and in purpose. According to Morris and Lambe (2014), 
MOOC videos are characterized by: (a) introductory videos in which course instructors explain 
the course and its purpose, (b) animations with audio narration in which the course content is 
explained, (c) video lectures given to real students, (d) documentary style video, (e) interviews or 
conversations among instructors and guests, and (f) video with built-in questions. A MOOC may 
make use of a particular video style or a combination of different styles according to its purpose 
of learning. 

Benefits of videos in online learning can be found in the literature (Triay, Sancho-Vinuesa, 
Minguillon, & Daza, 2016; Morris & Lambe, 2014). For instance, videos can be paused, repeated, 
or skipped through, allowing flexibility in the learning process and reinforcing student autonomy 
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in MOOCs. Short videos intertwined with quizzes emulate one-on-one tutoring and tend to fit into 
a manageable period of time that students can dedicate to MOOCs (Glance, Forsey, & Riley, 
2013). 

Although videos are the primary vehicle for delivering content in MOOCs, research seems 
to suggest that high quality videos are not enough to provide high quality experience to participants 
(Guo, Kim, & Rubin, 2014; Coetzee et ah, 2014). Lately, Lee et al. (2015) are investigating the 
potential of including time-anchored commenting interfaces along with videos that allow 
participants to watch the video and at the same time to exchange comments with others about the 
content being learned. This initiative has the potential to transform engagement with videos from 
passive to active. 

Student Engagement with Forums in MOOCS 

MOOC forums have many purposes. They can be a space for students to interact with other 
participants, get to know their peers, and learn through their experiences (Young, 2012). They can 
also be a space for students to check their understanding of the subject matter and to ask questions 
regarding a task or a problem (Young, 2012; Darabi, Arrastia, Nelson, Comille, & Liang, 2011). 
Forums are an environment for cooperation among students (Coetzee et al., 2014) in which 
students can learn as much from their interactions with others as they do from their interactions 
with course materials (Thomas, 2002). Forums can be a space where participants create new 
knowledge by interacting with others about concepts and techniques (Dubosson & Emad, 2015). 

According to Dubosson and Emad (2015), Young (2012) and Roller (2012), MOOC 
forums have been proven to be a good environment for peer assistance, in which students tend to 
answer each other’s questions without instructor intervention. However, research is unclear 
regarding the effectiveness/benefits of discussion forums. Some researchers found that discussion 
forums promote high quality discussion of course content, allowing students to reflect upon course 
materials and upon each other’s comments (e.g., Walker, 2007). Additionally, the implementation 
of discussion forums allows students the opportunity to initiate discussions and to drive their own 
learning (Darabi et al., 2011). In contrast, critiques of the use of discussion forums state that only 
a small fraction of participants contribute to forums, producing a high quantity of posts in relation 
to most participants (Coetzee et al., 2014). Threads may cover different topics, in different 
languages, showing problems of organization, which may intimidate participants and diminish 
their engagement (Anderson, Huttenlocher, Kleinberg, & Leskovec, 2014). As the number of posts 
and discussion threads become overwhelming, participants may feel less confident about engaging 
in forums (Dooley & Wickersham, 2007). In this sense, forums become spaces in which most 
students’ posts tend to display information acquisition instead of critical thinking (Kanuka & 
Anderson, 2007). For these authors, forums may support students’ increase of knowledge, but still 
fall short in presenting evidence as a venue for development of students’ new knowledge. 

Student Completion in MOOCs 

In open online environments such as MOOCs, participants are allowed to choose how they 
want to pursue their engagement. MOOC completion has emerged as an important metric being 
used by MOOC researchers to define course performance (Belanger & Thornton, 2013; Breslow, 
Pritchard, DeBoer, Stump, Ho, & Seaton, 2013). MOOC completion here is understood as 
completing course requirements and earning the Statement of Accomplishment certificate. The 
reason for choosing participant completion as a metric in MOOC is related to the straightforward 
way to collect information from the platform and completion being a variable that can be used to 
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compare students’ performance in different MOOCs (Wang & Baker, 2015; Moore & Kearsley, 
2011). However, when analyzing the literature of MOOC completion, the high levels of participant 
dropout emerge as a concern for MOOCs as a pedagogical environment, with completion rates 
varying between 5% and 15% as presented by Kizilcec and Schneider (2015), Ho et ah, (2014), 
and Jordan (2014). 

Due to the open environment of MOOC’s and participants’ autonomy of engagement, 
different participants may have different perspectives of what completion means. Our view is that 
completion in MOOCs gains a personal connotation that is aligned with participant’s goals in that 
MOOC. Loizzo et al. (2017) highlight that for some participants completion was related to 
acquiring new resources and interacting with the MOOC platform. McAuley, Stewart, Siemens, 
and Cormier (2010) also pointed out differences in participant views when defining MOOC 
completion. These authors noticed that as participants define completion they tend to do it based 
on their “learning goals, prior knowledge and skills, and common interests” (p. 4). On the other 
hand, as described by Anderson (2013), many participants who enroll in MOOCs do not have the 
intention of completing the course. This divergence of perceptions between MOOC providers and 
MOOC participants has led the field to explore alternative approaches to understanding MOOC 
completion (Koller, Ng, Do, & Chen, 2013). One of them is to focus on comparing students’ 
performance against their intentions in a MOOC (Kizilcec & Schneider, 2015). 

Student Assessment in MOOCs 

MOOCs offer regular opportunities for students to verify and test their understanding 
throughout the course. Thus, participants engage with non-graded quizzes as a way to test their 
knowledge, and at the end of the course they can elect to take a scored test for a course certificate. 
Student achievement in MOOCs is measured using assessments, which are mainly distributed into 
automated assessments (e.g. multiple-choice quiz), peer-assessment (e.g., students evaluate each 
other’s work), and self-assessment (students assess their own work). Since MOOC pedagogy is 
primarily based on mastery of learning (Glance et al., 2013) and there are a large number of 
students enrolled, the above approach seems to work well with MOOCs focused on serving a high 
volume of participants. 

As noted by O’Toole (2013), automated assessment may be used in situations where 
knowledge is fragmented into simple facts, algorithms, procedures, or explicit chains of reasoning. 
When knowledge is embedded in more complex situations, other forms of assessment may be 
applicable (e.g., peer-assessment). In a broad view, the literature on assessment indicates a need 
for online courses to create assessments for learning and not only assessments of learning 
(Admiraal, Huisman, & van de Ven, 2014). It is important to create assessments and feedback that 
are scalable, in which students can benefit from a reliable evaluation process with usable feedback 
that fosters opportunities for student learning (O’Toole, 2013). 

How Much Does Students’ Engagement in MOOCs Add to Their Achievement? 

Previous research indicates that student engagement in forums and watching of videos are 
related to their achievement in online courses (Kizilcec, Piech, & Schneider, 2013). The main 
purpose of the current study is to determine the effect of these two activities on student 
achievement in a MOOC. When analyzing student achievement in online learning, one must 
consider that achievement is related to a unique combination of the course characteristics and 
participants’ profile. This means that other MOOCs with similar course characteristics (videos and 
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forums) may not produce the same degree of student achievement as seen in the MOOC analyzed 
in this study. 

Thus, this study aims to answer the following research questions: (I) To what extent is 
students’ achievement in MOOCs associated with the number of posts made and the number of 
videos watched? (II) What effects does intention to certify have on student achievement in 
MOOCs? (Ill) What effects does intention to certify have on the number of videos watched and 
the number of posts made when considering student achievement in a MOOC? (IV) What 
cognitive activities are performed by participants as they interact in forums? 


Methods 

Study Context 

This study was conducted with students from the Creativity, Innovation, and Change (CIC) 
2.0 MOOC via the Coursera platform from July to August, 2014. The CIC MOOC was delivered 
over six weeks focusing on supporting students in achieving their creative potential, empowering 
them to transform their personal lives, organizations, and community. No pre-requisites were 
required to register for this course and students had to submit their work on a weekly basis for the 
full six weeks. Weekly lessons were comprised of students’ engagement with videos about 
explanations of new concepts and tools (e.g., CIC mindset and Intelligent Fast Failure), performing 
self-assessment upon their creative style, completing projects (e.g., shoes tower), readings, 
exercises, quizzes, and engaging in forums (Jablokow, Matson, & Velegol, 2014). For students 
who were interested in obtaining a certificate of completion, two options were offered. To receive 
a Statement of Accomplishment certificate, students had to submit six weekly tasks. To receive a 
Statement of Accomplishment with Distinction certificate, students had to complete the 
requirements for a Statement of Accomplishment certificate and submit 12 peer reviews 
throughout the course. 

Data Collection and Data Analysis 

Participation in this study was voluntary (N = 222), and recruitment was done upon 
students’ completion of a survey that explored the effect of groups on students’ success (course 
completion). Students of the CIC MOOC were invited to participate in an online survey that 
requested information regarding demographics, employment status, intention to complete the 
course, and preferred language. Data from the survey was used to gain knowledge about the 
characteristics of participants (demographics) and their intentions to complete the course. The data 
on number of videos watched, number of posts, and post content were retrieved from the Coursera 
platform. 

Students’ responses to the online survey along with their course interactions (i.e., number 
of videos watched and number of posts) were used in statistical analyses to build answers to 
research questions I, II, and III. Student’s forum posts were qualitatively analyzed using the 
Interaction Analysis Model (IAM) developed by Gunawardena et al. (1997) in order to build 
answers to research question IV. In answering the research questions I, II, and III, we created 
statistical models using stepwise binomial logistic regression. Age, gender, number of posts, and 
number of videos were the initial independent variables used to predict participants’ probability of 
achievement in the course. Participants were categorized into six age levels: 15-25 (25.87%), 26- 
35 (24.13%), 36-45 (18.53%), 46-55 (17.83%), 56-65 (9.44%), and 65 and above (4.20%). 
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Participants self-identified themselves as female (63.06%) and male (36.94%). Number of posts is 
comprised of participants (N = 222) posting new discussion threads and replying to each other’s 
posts. Number of videos watched represents all instances in which participants clicked the play 
button of a video in this MOOC. By design, data collection did not focus on participants’ 
engagement within videos such as pausing, fast-forwarding, and skipping video parts. Intention to 
certify were measured using a 5-likert scale 1-5 (strongly disagree - strongly agree). Participant 
achievement was coded as none, normal, and distinction, and later recoded as a binary variable 
indicating whether they achieved certification in the course. Next, we describe the Interaction 
Analysis Model used to build answers to research question IV. 

The I AM model (Table 1) was used as a tool to examine the cognitive activities performed 
by participants in their forums interactions. Data collection was comprised of all posts (450 posts) 
generated by the 222 participants across all forum categories (e.g., weekl, week2, etc.). The IAM 
model comprises five phases of knowledge co-construction that occur during the online 
discussions (Gunawardena et ah, 1997). They are: phase 1-sharing or comparing information, 
phase 2-discovery and exploration of dissonance or inconsistency among ideas, concepts or 
statements, phase 3-negotiating of meaning or co-construction of knowledge, phase 4-testing and 
modification of proposed synthesis or co-construction, and phase 5-agreement statement(s) or 
application of newly constructed meaning. Four hundred and fifty forum posts were analyzed 
according to the model. The unit of analysis was participant posts in the discussion forums. Each 
post was independently coded by the first and the third authors according to the level of cognitive 
activity (see column code , Table 1) and later checked for consistency and divergence. If any 
disagreement occurred in the coding, the code selected was the one based on a majority amount of 
evidence presented in the post. The inter-rater reliability was Cohen's Kappa = 0.99, indicating 
substantial degree of agreement between the two coders. 


Phase 

Operation which occur at this include 

Code 

PHASE 1 Sharing/Comparing of 

A. Statement of initial proposal, idea 

Phi/A 

information 

B. A statement of agreement from one or more other 
participants 

Phl/B 


C. Corroborating examples provided by one or more 
participants 

Phl/C 


D. Asking and answering questions to clarify details 
of statements 

Phl/D 


E. Definition, description, or identification of a 
problem 

Phl/E 

PHASE 2 The discovery and 

A. Identifying and stating areas of disagreement 

Ph2/A 

exploration of dissonance or 

B. Asking and answering questions to clarify the 

Ph2/B 

inconsistency among ideas, 

source and extent of disagreement 


concepts or statements 

C. Restating the participant's position, and possibly 
advancing arguments or considerations in its support 
by references to the participant's experience, 
literature, formal data collected, or proposal of 
relevant metaphor or analogy to illustrate point of 
view 

Ph2/C 


Table 1. Gunawardena, Lowe & Anderson’s (1997) Interaction Analysis Model 
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PHASE 3 Negotiation of meaning 
or construction of knowledge 

A. Negotiation or clarification of the meaning of Ph3/A 

terms 

B. Negotiation of the relative weight to be assigned Ph3/B 

to types of argument 

C. Identification of areas of agreement or overlap Ph3/C 

among conflicting concepts 

D. Proposal and negotiation of new statements Ph3/D 

embodying compromise, co-construction 

E. Proposal of integrating or accommodating Ph3/E 

metaphors or analogies 

PHASE 4 Testing and 
modification of proposed 
synthesis or 
co-construction 

A. Testing the proposed synthesis against "received Ph4/A 

fact" as shared by the participants and/or their culture 

B. Testing against existing cognitive schema Ph4/B 

C. Testing against personal experience Ph4/C 

D. Testing against formal data collected Ph4/D 

E. Testing against contradictory testimony in the Ph4/E 

literature 

PHASE 5 Agreement 
statement(s)/applications of 
newly constructed meaning 

A. Summarization of agreement(s) Ph4/A 

B. Applications of new knowledge Ph4/B 

C. Metacognitive statements by the participants Ph4/C 

illustrating their understanding that their knowledge 

or ways of thinking (cognitive schema) have changed 
as a result of the conference interaction 


Table 1 (cont.). Gunawardena, Lowe & Anderson’s (1997) Interaction Analysis Model 


Participants Demographics 

Participants who took part in this study (N = 222) came from all over the world. Table 2 
presents participants distribution and the countries where they were located. Data shows that 
Chinese participants accounted for the largest number of volunteers who participated in our study 
(21.6%), followed by participants from the United States (17.6%). This large number of Chinese 
students is related to the fact that the course has been translated into Chinese language. 


Country 

Number of 

Participants 

Country 

Number of 

Participants 

Country 

Number of 

Participants 

Country 

Number of 

Participants 

Country 

Number of 

Participants 

AR 

3 

CO 

4 

IR 

2 

PK 

3 

UK 

3 

AU 

5 

CR 

1 

KW 

1 

PT 

4 

US 

39 

AZ 

1 

DE 

1 

LC 

1 

RO 

3 

VE 

2 

BA 

1 

DK 

1 

LV 

1 

RS 

2 

ZA 

3 

BB 

1 

DO 

1 

MG 

1 

RU 

1 

ZW 

1 

BG 

1 

EG 

3 

MX 

10 

SA 

1 


BO 

1 

ES 

5 

MY 

3 

SD 

1 

BR 

6 

FJ 

1 

NE 

5 

SE 

1 

BW 

1 

GR 

2 

Nl 

1 

TN 

1 

CA 

10 

HR 

1 

NL 

4 

TW 

3 

CL 

1 

ID 

1 

NZ 

1 

UA 

1 

CN 

48 

IN 

18 

PE 

3 

UG 

1 


Table 2. Distribution of Participants by Country 
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The demographics indicate a sample of 140 females and 82 males. From a total of 222 
participants in this study, they classified themselves on a scale of four English levels as Poor 
(5.4%), Basic (22.5%), Fluent (41.4%), and Native (30.6%). Their age levels were classified into 
six ranges: 15-25 (30.6%), 26-35 (25.2%), 36-45 (16.6%), 46-55 (7.6%), and 66 and above (4.5%). 
In regard to their job status, 76 participants stated they are employed full-time (34.23%), 45 
participants were full-time students (20.27%), 37 participants were self-employed (16.67%), 17 
participants were employed part-time (7.66%), 16 participants were looking for a job (7.21%), 10 
participants were part-time students (4.5%), and 21 participants were retired, not working, or on 
maternity leave (9.46%). Regarding their intention to complete this MOOC, participants were 
asked to choose from a 5-likert scale 1-5 (strongly disagree-strongly agree) upon the statement 
“Intent to complete the course”. The distribution of participants’ answers to this survey question 
was: Strongly Disagree (3.15%), Disagree (6.70%), Neither Agree nor Disagree (54.30%), Agree 
(29.40%), and Strongly Agree (6.45%). It shows that in the pre-course survey, thirty-five percent 
of participants in this study indicated an intention to complete the entire MOOC. Participants’ 
course completion data was collected through Coursera with three levels of completion: none, 
normal, and distinction. These three levels of completion were recoded as a binary variable 
indicating whether they achieved certification in the course: complete (the combination of normal 
completion and completion with distinction) and non-complete. Participants’ achievement in this 
MOOC was distributed as: complete (41.52%) and non-complete (58.48%). On average, 
participants posted twice in forums and watched 40 videos throughout the course. The number of 
videos watched includes the possibility that some participants may have watched a video more 
than once. 


Results 

This section presents results of the analyses conducted to examine the extent to which 
student achievement in MOOC is associated with the number of posts made and the number of 
videos watched. It explores the effects of intention to certify on student achievement and on 
number of videos watched and number of posts made by students in MOOC. It also presents the 
cognitive level performed by participants as they interacted in forums. Statistical analyses were 
performed using the R programming language and the R-Studio Integrated Development 
Environment. Afterwards, discussion and conclusion are presented. 

To what extent is student achievement in MOOCs associated with the number of posts made 
and the number of videos watched? 

Results from binomial logistic regression model 1 (Table 3), presented both NofPost (p = 
0.0236) and NofVideoWatch (p = 9.98e-12) as statistically significant when considering 
participants’ achievement (Akaike Information Criterion, AIC = 220.7). Age and gender were not 
statistically significant regarding participants’ achievement. Parameter estimate for number of 
posts regarding the student achievement was 0.27, meaning that the number of posts is positively 
associated with student achievement. The 1.31 odds ratio for NofPost indicates that a one-point 
increase in the number of posts made is associated with the probability of MOOC achievement 
increasing by a multiplicative factor of 1.31 (Table 4). The estimate for frequency of videos 
watched on student achievement was estimated at 0.06 which informs that the number of videos 
watched is positively associated with student achievement. The 1.061 odds ratio for 
NofVideoWatch indicates that a one-point increase in the number of videos watched is associated 
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with the probability of MOOC achievement increasing by a multiplicative factor of 1.061 (Table 
4). In answering our first research question, students’ achievement in MOOC is positively 
associated with the number of posts they made and the number of videos they watched. 


Coefficients 

Estimate 

z value 

Pr(>\z\) 

(Intercept) 

-3.01187 

-4.963 

6.95e-07 *** 

Age 

-0.014502 

-1.221 

0.2222 

Gender 

-0.573418 

-1.572 

0.1159 

NofPost 

0.267124 

2.263 

0.0236 * 

NofVideo Watch 

0.059167 

6.807 

9.98e-12 *** 


Note: *p<0.1; **p<0.05; ***p<0.01 


Observations 


222 


Null deviance 

287.84 on 221 degrees of freedom 

Residual deviance 

210.70 on 217 degrees of freedom 

AIC 


220.7 


Table 3. Binomial Logistic Regression Presenting Number of Posts and Number of Videos as 

Statistical Predictors 





Coefficients 

Odds Ratio 

Confidence Interval (2.5%, 97.5%) 

(Intercept) 

0.04919961 

0.01411108 

0.1538637 

Age 

0.98560263 

0.96239826 

1.0085036 

Gender 

0.56359587 

0.27112845 

1.1397070 

NofPost 

1.30620194 

1.03895722 

1.6540583 

NofVideo Watch 

1.06095251 

1.04397637 

1.0803150 


Table 4. Odds Ratio of Binomial Logistic Regression Coefficients for Model 1 


What effects does intention to certify have on student achievement in MOOCs? 

Next, we added participants’ intention to obtain a course certificate as an independent 
variable in the logistic regression model 2 (Table 5). The results for NofPost (p-value = 0.02297), 
NofVideoWatch (p = 9.87e-12), and Certificatelntention (p = 0.00447) were statistically 
significant when considering participants’ achievement. The model improves when compared to 
the previous one, presenting a lower AIC = 213.45. 
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Coefficients 

Estimate 

z value 

Pr(>\z\) 

(Intercept) 

-5.502032 

-4.99 

6.03e-07 *** 

Age 

-0.009325 

-0.758 

0.44856 

Gender 

-0.61314 

-1.643 

0.10044 

NofPost 

0.278766 

2.274 

0.02297 * 

NofVideo Watch 

0.060217 

6.808 

9.87e-12 *** 

Certificatelntention 

0.535203 

2.843 

0.00447 ** 


Note: *p<0.1; 

**p<0.05; ***p<0.01 


Observations 


222 


Null deviance 


287.84 on 221 degrees of freedom 


Residual deviance 


201.45 on 216 degrees of freedom 


AIC 


213.45 



Table 5. Binomial Logistic Regression Presenting Number of Posts, Number of videos, and 
Intention of Certification as Statistical Predictors 


The odds ratio of binomial logistic regression coefficients for model 2 is presented in Table 
6. The 1.32 odds ratio for NofPost indicates that a one-point increase in the number of posts made 
is associated with the probability of MOOC achievement increasing by a multiplicative factor of 
1.32. The 1.062 odds ratio for NofVideoWatch indicates that a one-point increase in the number 
of videos watched is associated with the probability of MOOC achievement increasing by a 
multiplicative factor of 1.062. Likewise, the 1.708 odds ratio for certificate intention indicates that 
a one-point increase in certificate intention is associated with the probability of MOOC 
achievement increasing by a multiplicative factor of 1.7. Answering our second research question, 
model 2 shows that for the ones who intended to obtain a certificate, their probability of MOOC 
achievement increases by a multiplicative factor of 1.7 when compared to the ones who didn’t 
intend to receive a MOOC certification. Thus, the results above support the basic model that 
predicts student achievement in the CIC MOOC. In ANOVA test, model 2 presented a significant 
reduction of deviance (9.250) when compared to deviance in model 1 (75.799), as presented in 
Table 7. 


Coefficients 

Odds Ratio 

Confidence Interval (2.5%, 97.5%) 

(Intercept) 

0.004078477 

0.000412727 

0.03168829 

Age 

0.990718375 

0.966595199 

1.01460691 

Gender 

0.541647414 

0.255939155 

1.11289408 

NofPost 

1.321497502 

1.042604897 

1.69005426 

NofVideo Watch 

1.062066797 

1.044785161 

1.08180454 

Certificatelntention 

1.707795659 

1.200836587 

2.52256616 


Table 6. Odds Ratio of Binomial Logistic Regression Coefficients for Model 2 
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Models DfResid. Dev Df Deviance Pr(>Chi) 

Model 1: Achievement ~ Age + Gender + NofPost 

+ NofVideoWatch ' 217 210.7 2 

Model 2: Achievement ~ Age + Gender + NofPost 

+ NofVideoWatch + Certificatelntention 216 201.45 1 

_ Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 0,1 " 1 

Table 7. Analysis of Deviance for Models 1 and 2 

What effects does intention to certify have on the number of videos watched and the number 
of posts made when considering student achievement in MOOC? 

To understand the effects that intention of certification has on the number of videos 
watched and the number of posts made by MOOC students, we explored the logistic regression 
including interaction terms related to intention to certify and their moderation effects in students’ 
achievement (model 3). For the moderation effect of intention of certification, we used stepwise 
binomial logistic regression as presented in Table 8. Model 3 showed the best outcome when 
comparing results from other models (lower AIC). By including the interaction in the model, 
previously significant estimation of independent terms was reduced, and some independent 
variables became insignificant. 

Analyzing the predictors of model 3 in Table 8, only the interaction between “number of 
videos watched” and “intention of certification” was statistically significant (p-value = 0.0113) 
with odds ratio =1.019 (Table 9). Model 3 shows us that intention to certify does not play a 
moderating effect between the number of posts and student achievement. On the other hand, 
intention to certify has a moderating effect between the number of videos watched and student 
achievement. Answering our third research question, model 3 shows that an increase in 
engagement in videos for the ones who intend to receive a certificate is positively associated with 
an increase in their MOOC achievement. 


Coefficients 

Estimate 

z value 

Pr(>\z\) 

(Intercept) 

-1.813215 

-1.098 

0.2724 

Age 

-0.01057 

-0.829 

0.4069 

Gender 

-0.608227 

-1.572 

0.1158 

NofPost 

0.266914 

2.164 

0.0304 * 

NofVideo Watch 

-0.016493 

-0.545 

0.5859 

Certifi ca teln ten tion 

-0.356683 

-0.952 

0.341 

NofPost x Certificatelntention 

0.045786 

0.359 

0.71953 

NofVideoWatch x Certificatelntention 

Observations 

Null deviance 

Residual deviance 

AIC 

0.019166 2.535 

Note: *p<0.1; **p<0.05; ***p<0.01 

222 

287.84 on 221 degrees of freedom 

195.18 on 215 degrees of freedom 

209.18 

0.0113 * 


Table 8. Binomial Logistic Regression Presenting Number of Posts, Number of Videos, Intention 
of Certification and Interaction Terms as Statistical Predictors 


75.799 <2.2e-15*** 

9.25 0.002355** 
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Coefficients 

Odds Ratio 

Confidence Interval (2.5%, 97.5%) 

(Intercept) 

0.3262011 

0.004438981 

13.591961 

Age 

0.9901849 

0.965077567 

1.01487 

Gender 

0.550822 

0.252556101 

1.160282 

NofPost 

1.0156994 

0.385860806 

2.495412 

NofVideo Watch 

0.9821625 

0.924557622 

1.044336 

Certificateln ten tion 

0.587316 

0.234989466 

1.586455 

NofPost x Certificatelntention 

1.0647178 

0.858939199 

1.342143 

NofVideoWatch x Certificatelntention 

1.0196866 

1.004576628 

1.035409 


Table 9. Odds Ratio of Binomial Logistic Regression Coefficients for Model 3 


Synthesizing the results from our first three research questions, this study showed that 
student achievement in MOOCs is positively associated with the number of posts made and the 
number of videos watched. From model 2 we learned that a one-point increase in NofPost is 
associated with the probability of MOOC achievement increasing by a multiplicative factor of 
1.32, and a one-point increase in NofVideoWatch is associated with the probability of MOOC 
achievement increasing by a multiplicative factor of 1.062. For students who intend to obtain a 
certificate, their achievement in the MOOC increases by a multiplicative factor of 1.7 when 
compared to the ones who do not intend to obtain a course certificate. We also learned that 
intention to certify does not play a moderating effect between the number of posts and student 
achievement. On the other hand, intention to certify has a positive moderating effect between the 
number of videos watched by students and their achievement in this MOOC. In the next section, 
we qualitatively analyze participants’ posts to depict the cognitive activities performed by them 
through their interactions in forums. 

What cognitive activities are performed by participants as they interact in forums? 

We used the Interaction Analysis Model (IAM) from Gunawardena et al. (1997) to examine 
the cognitive activities performed by participants in their forums interactions. Results from 
qualitative coding on 450 forum posts are shown in Table 10. Participants’ forum posts were coded 
through the IAM’s levels from phase 1 to phase 3 (see Table 10). No forum posts were located 
under phase 4 (testing tentative constructions) and phase 5 (statement/applications of newly 
constructed knowledge) of the IAM. Data shows that the majority of participants’ posts were 
situated on making statements of observations related to the course and statements showing 
participants’ opinions, as represented under the code Phl/A comprising 72.07% of the data. This 
prominent characteristic of participants’ posts may be understood by looking at the forum design 
of the CIC MOOC. In the forums, it was recommended that participants post to others by using 
the P.U.R.E approach of giving feedback. According to the CIC MOOC page 
(https://www.coursera.org/learn/creativity-innovation) , the P.U.R.E. approach stands for Positive, 
Unique, Revision and Education as described in Table 11. 

Analyzing the P.U.R.E. framework recommended to CIC MOOC participants as they 
interact with others in forums, we noticed that the suggested script enhances participants’ 
collaboration, minimizing conflict among them. According to Gunawardena et al. (1997) in 
discussions where there is little conflict among participants’ ideas, participants tend to “accept 
each others’ statements or examples as consistent with what the group members already know or 
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believe and the discussion may never advance out of phase one” (p. 415). Another factor that may 
have contributed to participants’ lack of higher levels of cognitive activities might be related to 
the nature of the posts in this sample. Although data collection comprised all posts from the 222 
participants across all forum categories (e.g., weekl, week2, etc.), when content of these posts was 
qualitatively analyzed through the I AM we noticed that many of these posts’ content resembled 
the structure of participants’ initial posts in MOOCs, in which they typically introduce themselves 
and state their goals and expectations for the course without challenging ideas of each other. For 
example, “Hello I'm Jaime [pseudonym]. I'm a software engineer in Taiwan. I like innovative 
products and want to know more about creativity. Cheers, Jaime.” This piece of evidence leads us 
to hypothesize that these participants may have had a higher engagement at the beginning at the 
course. 


Low mental level 

High mental level 


Phases 

Phl/A 

Phl/B 

Phl/C 

Phl/D 

Phl/E 

Ph2/A 

Ph2/B Ph2/C 

Ph3/A 

Ph3/B 

Total 

#of 

codes 

387 

22 

11 

76 

24 

10 

1 3 

2 

1 

537 

%of 

codes 

72.07% 

4.10% 

2.05% 

14.15% 

4.47% 

1.86% 

0.19% 0.56% 

0.37% 

0.19% 

100.00% 


Table 10. Participants’ Posts Analyzed According to the Interaction Analysis Model (IAM) 
(Gunawardena et al, 1997) 


Positive 

First, highlight the strengths of the submission - what was done well? Give “happy comments” 
right away, and be considerate and constructive 

Unique 

Next, comment on interesting and unique aspects of the submission - what did you find 
interesting about it? 

Revision 

Third, be a “critical friend” - what suggestions do you have for improvement or revision. The 
purpose is not so much “to grade” as to interact and suggest. 

Education 

Close by reflecting on what you learned and how you were educated from the submission as a 
feedback provider. 


Table 11. P.U.R.E Approach of Giving Feedback Used in CIC MOOC 


Qualitative analysis of participants' forum posts showed that they tend to post in the format 
of statement of opinion followed by a question. For example, “I'm struggling with creative blocks. 
I think part of it is fear of failure but what are some strategies to get your creative juices flowing?” 
This intrinsic characteristic of the nature of participants’ posts justifies the high quantity of Phl/A 
and Phl/D codes in Table 10. In analysis of CIC MOOC posts, participants also engage with others 
in forums by making a statement of agreement with other participants’ posts (Phl/B) and 
corroborating with ideas of others (Phl/C), for example, “Hello Sidney! [pseudonym] As I reflect, 
I agree that [the] shoe tower exercise is simple but profound. As you head into your new venture 
it will help you to innovate and build from small failures and challenges along the way.” 

In terms of disagreement and negotiation of meaning (phases 2 and 3 of the IAM), 
participants' posts were mainly concentrated in Ph2/A in which they stated potential 
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disagreement/divergence with the MOOC’s rules and/or with the MOOC’s activities, as in the 
example: 

I’m disappointed in this choice of exercise. I work with people who own one pair 
of shoes—actually flipflops—at a time. The choice of shoe tower reflects a cultural 
myopia that has me worried about the rest of the course... 

This participant was referring to the task presented in week 2 of the CIC MOOC in which 
students had to build the tallest shoe tower without using any external support. The shoe tower 
task was designed to foster participants’ creativity dealing with limited resources (available 
number of shoes) and to help them reflect about the strategies used in their creation process. In the 
data sample, only two posts presented Ph3/A level and only one post presented Ph3/B level, in 
which the participant intended to clarify meaning and the weight assigned by another participant 
to the definitions of the word “bad.” Participants’ posts in this study did not show tendency to 
disagreement. In fact, in most of the posts they tended to be polite and friendly, which may reduce 
situations of dissonance and disagreement as the ones suggested in phase 2 of IAM and beyond. 


Discussion and Conclusions 

Using data from volunteer participants in the CIC MOOC, this study investigated the extent 
to which students’ achievement in MOOCs is associated with the number of posts made and the 
number of videos watched. Results indicate that participant engagement in forums has a bigger 
impact on the probability of MOOC completion (32%) when compared to the contribution of 
videos watched (6%). Considering direction of coefficients, intention to certify had an amplifying 
effect on students’ achievement, acting as a moderator in enhancing students’ achievement. These 
results are aligned with the literature on MOOCs in which forum participation supports students’ 
completion of the course (B reslow et al., 2013; Kizilcec et al., 2013; Waldrop, 2013; Daniel, 2012). 

Qualitative analysis revealed that the forum contributions from volunteers in this study are 
mainly located in phase 1 of Gunawardena’s et al. (1997) Interaction Analysis Model. This 
empirical result adds evidence to the literature of forums (Kanuka & Anderson, 2007, Thomas 
2002) that state that most students’ posts in forums tend to display information acquisition instead 
of critical thinking. In this sense, forums in this study worked as a venue in which students 
increased knowledge about others and about the course content, but fell short in presenting 
evidence as a venue in which students develop new knowledge. 

Given the importance of participants’ engagement with forums and videos in MOOCs, 
implications for practice suggest MOOC designers and instructors should create discussion 
prompts that foster interactions about deep meaning of concepts or application of concepts. 
Knowing that participants in MOOCs tend to engage in forums by posting under the format 
“statement” plus “question,” MOOC instructors can capitalize on this common type of post to 
foster participant replies to posts of others. In doing so, this participant behavior may lead to the 
creation of a community of learning in which more knowledgeable participants answer the queries 
of novices or less knowledgeable individuals. On the other hand, if a forum presents only the 
behavior of participants asking questions but not replying to each other, it may incur the risk of a 
forum becoming a pile of posts as described by Thomas (2002). 

In terms of videos, this study extends the literature showing the impact of videos in the 
probability of MOOC completion. MOOC designers can use the results of this study as a rationale 
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to suggest the implementation of more interactive videos as described by Glance et al. (2013), 
intertwining segments of videos with quizzes, and improving the descriptions of videos so that 
participants know exactly what they can learn by interacting with that specific video. Although it 
is hard to control participants’ engagement with videos in MOOCs by design, providing better 
guidance of what videos may contribute to participants’ learning paths may help them to make 
better decisions regarding engagement. 

In terms of research, although many may intuitively believe that student engagement with 
forums and with videos seems to be related to achievement, this study advances the field by 
showing how much these engagement types have the potential to affect student achievement. The 
combination of statistical methods and qualitative analysis provide to the literature a model of how 
to analyze student engagement in MOOCs. Results presented here support students’ achievement 
in heterogeneous student populations, as the ones presented in MOOCs. By understanding the 
potential contribution from students’ engagement with forums and with videos in their course 
achievement, new approaches can be developed to create and sustain learning through 
personalization. Knowing features that affect achievement and how much achievement is affected, 
MOOC designers can develop learning paths aimed at maximizing participants’ completion of a 
MOOC. 

Limitations and Future Work 

The statistical analysis presented in this study makes use of a case study survey and click 
data which don’t afford explanations regarding the causes and effects among the variables. Thus, 
our results regarding participants’ intention to complete the MOOC are contingent on CIC MOOC 
students’ honesty and disposition to volunteer in this study. The sample size of this study was 
relatively small when considering the large number of students who enrolled in this MOOC. Due 
to size effects, the findings may not be generalizable to other samples. This could be overcome 
with the implementation of longitudinal studies in which multiple MOOC cohorts are analyzed 
under the same statistical model and under the same qualitative framework. Knowing the potential 
impact on students’ achievement as they engage with forums and with videos, in future work it 
will be interesting to study how the patterns found from the Interaction Analysis Model informs 
participants’ learning outcomes. 
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